DOCUMENT RESUME 



ED 477 693 



SE 068 067 



AUTHOR 

TITLE 

INSTITUTION 
PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Underwood, Jody S.; Hoadley, Chris; DiGiano, Chris; Stohl, 
Hollylynne; Hollebrands, Karen 

Design Principles of the ESCOT Math Environments. 

Educational Testing Service, Princeton, NJ. 

2003-04-00 

19p.; Paper presented at the Annual Meeting of the’ American 
Educational Research Association (Chicago, IL, April 21-25, 
2003) . 

Reports - Descriptive (141) — Speeches/Meeting Papers (150) 
EDRS Price MFOl/PCOl Plus . Postage . 

^Computer Software Development; ^Computer Uses in Education; 
Educational Environment; Educational Principles; Educational 
Technology; Elementary Secondary Education; ^Science 
Education; ^Student Centered Curriculum 



ABSTRACT 

This paper describes the Educational Software Components of 
Tomorrow (ESCOT) project". The focus of the project was on principles that 
support problem-solving and learner-centered design issues, and the purpose 
was to garner lessons from a large educational software development project 
to share with the learning sciences and other interested communities who 
develop learner-centered software. The Identifying Design Principles in 
Educational Applets (IDEA) project, background of ESCOT, the Math Forum’s 
Problems of the Week.(PoWs), data mining for design principles, and design 
principles are presented. (KHR) 




Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Ob&ok 1- 



Educational 
Testing Service 



Distributed at AERA, April 2003 




Design Principles of the ESCOT Math Environments 



Jody S. Underwood, Educational Testing Service 
Chris Hoadley, Penn State University 
Chris DiGiano, SRI International, Center for Technology in Learning 
Hollylynne Stohl, North Carolina State University 
Karen Hollebrands, North Carolina State University 



Paper presented at the annual meeting of the American Educational Research Association (AERA) 

April 21 to 25, 2003, Chicago, IL. 



Copyright © 2003 by Educational Testing Service. All Rights Reserved. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

__ document has been reproduced as 
received from the person or organization 
originating it. 



□ Minor changes have been made to 
improve reproduction quality. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 




BEST COPY AVAIMBLE 



2 



I Educational 
i Testing Service 



Distributed at AERA, April 2003 



1 Introduction 

The Internet is increasingly becoming a vehicle for various authors to publish educational 
software in the form of miniature applications (e.g., applets). The Educational Software 
Components of Tomorrow (ESCOT) project is an example of one such effort. After 
forging ahead with designing, publishing, and utilizing mathematics applets with many 
middle school students, we had a large database to draw upon for Identifying Design 
Principles in Educational Applets (IDEA). To begin the process of evaluating this 
existing library of educational software for mathematics, the authors of this paper focused 
on design principles that successfully support problem solving by drawing on such data 
as videos of students using the software and summaries of written student work. The 
focus of this project was not on user interface design specifically, but on principles that 
support problem-solving and learner-centered design issues. The purpose was to gamer 
lessons from a large educational software development project to share with the learning 
sciences and other interested communities who develop learner-centered software. 

Each piece of ESCOT software was accompanied by a context, a set of questions, and a 
Java-based applet (sometimes more than one applet) to help students answer the 
questions. They were posted to the public as part of the Math Forum’s Problem of the 
Week (PoW) (http://mathforum.org/pow/) during 1999-2001, and called ESCOT PoWs 
(EPoWs). 

Our approach to generating the design principles was to select a subset of the 42 EPoWs, 
for which data held the most promise for analysis (e.g., EPoWs that were revised from 
year 1 to year 2, a large number of student submissions). We then collected our expert 
opinions and built consensus ratings about each of the selected EPoWs. From that, we 
generated design principles, refined them, and categorized them. Each design principle 
has an associated intended effect for the user. We did empirical validation and further 
refinement of the design principles and of the intended effects by watching videos of 
students using the EPoWs. The resulting design principles fall into four categories (see 
Table 1). 

Five of the authors were participants in ESCOT, each with different areas of expertise - 
middle school teacher, software developer, educational technologist, mathematics 
educator, and project evaluator. The remaining authors, not having been part of ESCOT, 
brought objective views about the software we set out to evaluate, and had 
complementary areas of expertise - teacher, mathematics educator, and technology 
designer. 

The IDEA project’s objectives were to identify principles that can guide the design of 
effective learning technologies through the analysis of applets created by the Educational 
Software Components of Tomorrow (ESCOT, NSF-funded) project, which is briefly 
described below. This work will contribute to the Center for Innovative Learning 
Technologies (CILT) Design Principle database [http://wise.berkeley.edu/design/] by 
adding design principles based on 25 concrete cases. We are hoping that with so many 
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applets being analyzed against student data that a number of lessons can be learned about 
the design of powerful educational technologies. We intend to share our findings with the 
learning sciences community and other relevant communities. We regard these design 
principles and their validation as a starting point in a conversation. We hope that others 
will continue this conversation through the web site that houses the design principles as 
well as in other forums. Anyone interested in educational software design or the 
selection of educational software may find these design principles useful. 



Table 1. Sample design principles, their categories, and intended effects. 



Category 


Sample Design Principle 


Intended Effect 


Motivation 


Enable early reward for students (e.g. provide 
easy questions or activities they can do 
successfully) 


get involved in the problem that 
leads toward producing a solution 


Presentation 


Make links between representations obvious and 
imgratuitous 


less division of attention, 
understanding relationships 


Support for 
problem-solving 


History of actions 


can lead to reflection, strategy 
tuning, and not wasteful 
duplication 



2 The IDEA project: Connection to Theory 

We began the IDEA project with the goal of extracting valid design principles from the 
ESCOT experiences. As researchers in instructional systems design have noted, what 
“valid” means may vary widely depending on what the principles are for — explanation 
and prediction or supports for the work of expert designers (Reigeluth, 1999). Should 
they be judged on their weaknesses (when they fail to work) or their strengths (when they 
are usefol, regardless of whether they are comprehensive and “true”) (Snelbecker, 1 999). 
As George Fox once noted, “All models are wrong, but some are useful.” Our goal in 
undertaking this project was to focus on the utility of these design principles while 
insisting on the highest degree of “truth” we could adduce — specifically, the principles 
had to be consistent with all the data we had at hand. 

Simon (1969) defined design knowledge as a “science of the artificial” and highlighted 
that design science, in contrast to natural science, involves models that are relative not 
only to that which is designed (the artifact itself) but also, fundamentally, to the artifact’s 
relationship to the setting and to external goals. It is in this spirit that we set out to learn 
what we could from the ESCOT experiences. In particular, we are highly aware that since 
we were doing an analysis of applets used in a very particular setting (the Problem of the 
Week context, in which the designers had little control over the context of use), we would 
only be able to make tentative models of how the applets influence problem solving. 

While we only had one context in which to examine the interventions, we did have, in 
some cases, iterations on those interventions from Year 1 to Year 2 of the ESCOT 
project, and considerable access to the rationale for both the original versions of the 
EPoWs and successive iterations, in addition to the designers’ own assessments of the 
success of their endeavors. Our goal, then, was to do a “post mortem” — to examine the 
principles that drove the design process and our post hoc interpretations of the outcomes, 
and to uncover the principles that we felt would be most useful. 
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Our goal was that the principles we uncovered would be ones with which we had some 
direct experience in the EPoW context, that either motivated the powerful successes or 
explained the clear failures, and that were consistent with the data at hand. In this sense, 
our work is like a design experiment (Brown, 1992; Collins, 1992), with one important 
exception: the extraction of design principles was done after the fact, and data was not 
collected during deployment for the explicit purpose of testing these design theories. We 
suspect that this sort of post-mortem analysis may be an important design-based research 
method that differs from the design experiment tradition. (Design-Based Research 
Collective, 2003; Hoadley, 2002) 

We hope that these principles will be taken as objects for further discussion, examination, 
and eventually for adaptation in the design of other interventions, in the flexibly adaptive 
design sense (Schwartz, Lin, Brophy, & Bransford, 1999) with explicit acknowledgment 
that there is room for adaptation without “lethal mutation” (Brown, 1992), and with the 
knowledge that “your mileage may vary.” In short, these principles are our best attempt 
to derive the most supported hypotheses from our data that deserve further examination 
by others in other settings. 

Typically, design principles arise in one of two ways: either as a transcription of known 
techniques or strategies that have arisen through long experience (so-called “craft 
knowledge”), or through a combination of deduction from and extrapolation from known 
scientific theories of systems. Both rest on an empirical base, but in vastly different ways. 
For instance, an experience-based design principle in architecture may say that “thatched 
roofs made of wheat straw may be indefinitely and effectively maintained by removing 
damaged top layers and leaving the old base layers.” (Hohle, 2003) This principle, based 
on the craft traditions of centuries (indeed, millennia) of roof thatchers in England, arose 
through experience with thatched roofs in the British climate. In contrast, a theory-driven 
architectural design principle might suggest that aluminum plumbing is preferable to iron 
or copper because it does not corrode as easily, based on (empirically-derived) theories of 
chemistry. However, both approaches have limitations. In the first case, the principle may 
have great weight of prior evidence, but if conditions change (as with the spread of wheat 
hybridized for food in Britain) the principle may fail (Hohle, 2003), and the lack of a 
mechanism or theory undergirding the principle may seriously hamper knowing when or 
why the principle will work. Similarly, theory-based design principles such as the 
aluminum-pipe principle may fail due to pragmatics in the situation that are not evident in 
the theory, such as the difficulty of joining aluminum piping systems to other metals 
without causing electrical currents which corrode both, or health concerns about 
aluminum in drinking water. This suggests that, ideally, principles should answer both to 
tradition and to theory, in addition to possibly being empirically tested in studies 
specifically designed to falsify them if they are untrue. We used a process that was both 
inductive (like craft knowledge) and deductive (like theory-generated knowledge) for 
selecting the most plausible design principles from the ESCOT experience. This process 
is described in detail below. 
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3 Background: ESCOT, the Math Forum, & PoWs 

ESCOT was a very large, coordinated effort sponsored by the National Science 
Foundation. The project was led by SRI International in Menlo Park, CA, with key 
subcontractors at the Math Forum (now located at Drexel University); the University of 
Colorado, Boulder; the University of Massachusetts, Dartmouth; KCP Technologies, Inc 
(in Emeryville, CA); and the University of Missouri. All together, we had 7 major 
partners, and 34 organizations that contributed software, volunteers, and/or contractors. 
Key to the ESCOT network was a cadre of mathematics teachers who participated in 
design teams. A total of 19 teachers were involved, hailing fi-om 1 1 different states. 

The project title refers to “component software,” and, indeed, much of our work revolved 
around a technical vision of modular, mix-and-match software tools (Roschelle et al., 
1999). From these component tools, we developed a series of ESCOT Problems of the 
Week (EPoWs) in support of middle school mathematics. Over the course of two school 
years, integrated design teams, consisting of professional programmers, teachers, and 
educational technologists, produced 42of these EPoWs on a regular schedule. Each 
EPoW consisted of a motivating story, a set of driving questions, and an applet to help 
students solve the problem and answer the questions. 

The Math Forum provided a context for exploring what would constitute a viable 
technical infi^astructure for the development of interactive software to support student 
learning. In the Math Forum’s well-established Problems of the Week (PoWs; 
http://mathforum.org/pow/) . students read the problem, work on a solution either 
individually or with a peer or group, and write an explanation of how to obtain a solution. 
The students submit their solutions with explanations, receive feedback about their work, 
and are encouraged to submit a revised solution if there are areas that can be improved. 
The EPoWs were designed to follow this same arrangement. 

The EPoWs were developed over two academic years. Student submissions were and 
continue to be part of the Math Forum archive (http://mathforum.org/escotpow/) . 

Findings from Renninger et. al.’s (2001) study of student work with these problems 
suggests that students’ competence and achievement account for differences in their 
problem solving. However, the computer-based learning environment appeared to 
override differences that would typically be found as a function of interest and self- 
efficacy with respect to students’ abilities to connect to, generate strategies for, and be 
autonomous in problem solving. These differences may be due to the design of the 
EPoWs. 

When the ESCOT project ended, it left a legacy of a large amount of data that could be 
mined. After the EPoWs were published on the Web, hundreds of student solutions were 
captured on the Math Forum web site; sessions of some students using the EPoWs were 
video taped; and design decisions made by the teams who created the EPoWs (see figure 
1). These data informed the current effort of identifying design principles for educational 
software. 
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Figure 1. ESCOT Problem of the Week data 



4 An Example EPoW: Fish Farm 

During the second year of ESCOT (2000-2001), the Fish Farm EPoW (see Figure 2) was 
developed. It was purposely designed to give students access to different possible 
solution strategies and representations for making sense of the problem. This problem is 
open-ended in the sense that there are multiple strategies to use and three possible correct 
solutions. Although this problem can be solved using algebraic techniques, the intent of 
using the problem situation and tools in the java applet was to engage students in thinking 
about different strategies and solution paths, as well as part-part and part-whole reasoning 
and equivalent ratios. The bonus question was designed to induce a perturbation for 
students about the relationship between a part-part and a part-whole representation of a 
ratio. The students are asked to compare the part-part ratio of 1 :2 to a pie graph showing 
apart-whole l/3:2/3 representation. Many students intuitively think about a 1:2 ratio as 
representing a one-half situation and do not easily make the transition to a l/3:2/3 
representation. 

The applet was created with a tank on its left side with 26 fish (13 males, 13 females) that 
the user could “drag and drop” into one of the three ponds to its right. As a fish is 
dropped into a pond, a numerical count and pie graph are updated to keep tally of the 
number of females and males and the percent of females and males in each pond. Once a 
fish is “dropped” into a pond, it will swim within the boundaries of the pond. The RUN 
button at the bottom of the screen is used to activate the applet so that the “updates” and 
“swimming” occur when a fish is dropped in a pond; however, a user can move the fish 
without hitting the RUN button. The STOP button deactivates the “updates” and 
“swimming” features. The RESET button will place all 26 fish back into the tank on the 
left, while the CLEAR button will erase all fish from the applet. 
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A Fishy Family: For their birthday, the Carp triplets 
received 26 tropical fish: 13 females and 13 males. 

They discussed ways to divide the fish among then- 
three tiny backyard ponds. 

Angel said, "I want the same number of male and 
female fish in my pond." 

"Okay," said Molly. "I want three times as many 
males as females in my pond." 

"Then I want twice as many females as males in 
my pond," Gar replied. 

Is there a way to put all 26 fish into those three 
ponds, while giving each triplet what he or she 
wants? Use the applet to explore this question. 

Questions: 

1. How many male fish and female fish does each triplet get in his or her pond? Describe the work you 
did to find the solution. Sample questions you can answer: Into which pond did you put fish first? 
How many fish of each kind went into that pond? Why? What was your next step? How were you 
sure a pond had the correct ratio? 

2. Given the 13 males and 13 females, what are ALL the possible numbers of male and female fish that 
would satisfy the ratio of 1 male to 2 female fish in Gar's pond? Explain why these different amounts 
are equivalent to the ratio 1:2. 

Bonus: Explain why all possible answers in question 2 result in the same pie graph for Gar's pond. 

Figure 2. Fish Farm EPoW. 




1 If^ ; 1 UP 



fArrttywanls 3 ]H^ : 1 up 
^1^ Him' are; 3)1^ 1 

C-M wmtn 1 Xii : P 






Interactive Java Applet 



The complete materials associated with this EPoW include (available online at 
http ://mathforum. or g/escotpow/ so lutions/ solution. ehtml?puzzle=40) : 1) the problem 
situation and questions, 2) an interactive applet, 3) a teacher support page with 
suggestions for pre and post activities, and 4) expected solutions. The solutions were 
prepared by the design team and used by mentors who provided feedback in response to a 
student’s solution. After solutions were no longer accepted for an EPoW, it was archived 
along with comments from a lead mentor summarizing students’ solution strategies, 
difficulties, and sample student responses. 



Below is part of one 13-year-old’s solution to the first question in the Fish Farm EPoW. 
Notice that the student described her strategy in terms of ratios, and used arguments 
about the properties of mathematical operations with ratios to justify her approach. 

Angel had 8 male fish and 8 female fish in her pond. Molly had 3 male fish and 1 
female fish in her pond. Gar had 2 male fish and 4 female fish in her pond. I first put 
one male and one female in Angel's pond, 3 male and 1 female in Molly's pond, and 2 
males and 4 females in Gar's pond. I thought I could put the rest into Angel's pond. 
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but I noticed that there was unequal amounts of males and females. So to make it 
equal, I put one more male fish and 2 more female fish in Gar's pond, that would still 
be the same as 1:2. That left me with the same amount of male fish and female fish, 
so they could all go into Angel's pond (that would still equal 1:1). Since I did 
everything slowly, I made sure that my amounts of fish were equal to the ratios. All I 
did was get the total amounts and then reduce them, and the reduced number 
should've equaled the ratio. I knew I got everything right when the bricks turned 
green. 

From this student’s description, it appeared that several of the design elements in the 
applet provided tools for her to complete the task. Specifically, the displayed ratios 
allowed her to compare the pond’s male-to-female ratio with the desired ratio so she 
could check if one ratio reduced to the other. It appeared that the bricks turning green 
also provided closure and confirmation that her solution was correct. This type of 
interaction and response was important for us to consider as we mined the EPoW data to 
identify design principles and intended effects of those principles. 

5 Data Mining for Design Principles 

5.1 Phase I: Expert Opinion 

Design principles (DPs) were identified for the EPoWs using a six-step process. First, a 
subset of the EPoWs was selected for mining. Second, all the selected EPoWs were 
reviewed and a preliminary set of design principles was identified. Third, a popularity 
contest of the EPoWs was done to see how closely the experts agreed on their values. 
Fourth, design principles were generated by a consensus building approach. Fifth, the 
appearance of the preliminary set of design principles in each of the twenty- five EPoWs 
was noted. Sixth, the design principles were categorized into ease of applet use, 
motivation, presentation, and support for problem solving. 

1. EPoW selection. Of the 42 EPoWs that were developed for the ESCOT project, we 
narrowed the bank to draw on for the purpose of mining design principles. Two key 
points were considered: 1) there were at least 20 submitted student solutions, or 2) the 
EPOWwas part of a l®*-2"‘^ year generation pair. The 25 EPoWs that were mined can 
be found in Appendix A. 

2. EPoW review. We followed a top-down approach to identifying the design 
principles. We, as “experts” from a variety of disciplines and perspectives, reviewed 
each selected EPoW, noted what characteristics we valued with respect to its 
mathematical purpose. We also reviewed the design rationales from the design teams 
and mentor summaries of students’ typical solutions and strategies for each EPoW. 
These were the building blocks from which the design principles were generated. 

3. Overall Quality Ranking. As a starting point for our discussion of principles for the 
design of high quality applets, we individually ranked the overall quality of the 
selected EPoWs. We each brought different expertise to the table, which informed our 
perceptions of the EPoWs. The ranking exercise helped bring these differences of 
opinion to the fore. The activity also initiated a valuable discussion on how individual 
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applet features combined to give an overall impression. We then took the individual 
rankings and, after some discussion, aggregated them into a single list of rankings. 
The EPoWs and their quality rankings are listed in Appendix A. The highest ranked 
EPoWs were Galactic Exchange, Search and Rescue (year 1), and Fish 2. The lowest 
ranked EPoWs were Scale and Bowl, Rock Paper Scissors 1, Fractris, and Marabyn. 

4. Design Principle generation and consensus building. At this point, we combined 
the lists of design principles that we had generated in the small groups. We then 
reviewed this larger list, combined and came to consensus on definitions, and reduced 
the list of design principles to ones that were evident in several EPoWs. 

5. DP appearance in the EPoWs. We took the list of principles that were found in a 
subset of high and low ranking EPoWs and noted which EPoWs followed and 
violated each. Small groups consisting of 2-3 researchers took each of the design 
principles and noted where they appeared and did not appear in each of the EPoWs. 
Each of the small groups did this for a common set of EPoWs, coming to consensus 
on the definitions of the DPs. Once we reached consensus, we did the mapping onto 
the additional EPoWs. 

V. 

6. DP categorization. The design principles were categorized into ease of applet use, 
motivation, presentation, and support for problem solving. We decided that the ease 
of use DPs encompassed principles of user interface; thus we subsequently focused 
our efforts on the later three categories. These are described in more detail in the 
Design Principles section. 



5.2 Phase II: Empirical validation 

Once design principles and intended effects were hypothesized, videos of students 
interacting with some of the EPoWs were examined. The videos were sampled to include 
the following: 

1 . Two high ability 8*'’ grade boys using Fish Farm 1 

2. Two lower ability 8*'’ grade girls using Fish Farm 1 

3. Two low ability 8*'’ grade students (one boy and one girl) using Fish Farm 1 while 
working with a pre-service mathematics teacher 

4. Two lower ability 8*'’ grade girls using Scale n Pop 

These videos were original data sources collected from several research projects 
conducted by various members of the ESCOT team, including the Renninger et. al. 

(2001) study of student work with the EPoWs and the Stohl research program (Stohl, 
2003; Stohl, under review). In each video, students were encouraged to think aloud while 
working in pairs at a computer with access to paper and pencil. The video images were 
captured in such a way to observe both verbal and non-verbal information about how the 
students interacted with the EPoW. The ability to observe students directly interacting 
with the applets was useful for coding. 
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Our intention for examining the videos was to locate evidence to support whether or not a 
design principle (DP) was followed, and whether or not an intended effect (IE) occurred. 
Thinking about these options as a 2x2 matrix allows us to see four possible outcomes (see 
Figure 2). 



Table 2. Coding for student videotape data. 





Design Principle Followed 


Design Principle Violated 


Evidence of Intended 
Effect Displayed 


FE (followed, with effect) 


VE (violated, with effect) 


No Evidence of Intended 
Effect Displayed 


FNE (followed, no effect) 


VNE (violated, no effect) 



These four outcomes (FE, FNE, VE, and VNE) were used as we coded video segments. 
Each researcher had a chart that listed each DP, IE, and space for recording descriptions 
of segments from the video (including timestamps) that provided evidence supporting 
(FE and VNE) and evidence against (FNE) a design principle (see Table 3 for examples). 
For the fourth case, when a DP was violated but there was evidence that an IE was 
achieved (VE), no conclusion could be made about whether the evidence supported or 
refuted a design principle. Thus, a segment coded as VE was inconclusive and not used to 
support to refute a DP and IE. 



Table 3. Examples of evidence of design principles in several video sessions. 



Category 


Design Principle 


Intended Effect 


Evidence 


Video Session 


Motivation 


Enable early reward 
for students (e.g. 
provide easy 
questions or 
activities they can 
do successfully) 


get involved in the 
problem that leads 
toward producing a 
solution 


FE (min 5) They were happy when the 
balloon released. 

FE (min 17-18) They were happy when the 
balloon enlarged for the improper fraction 
booth. 


Scale n pop, 
two girls 


Presentation 


Links between 


less division of 


FE (min 11-13) One girl knew to use the 


Fish 1, 




representations 
should be obvious 
and ungratuitous 


attention, 

understanding 

relationships 


sums instead of counting the fish. 

FNE (min 11-13) The other girl didn't know 
to use the sums. 

VNE (min 3-6) The girls expected the other 
representations to be updated when they did 
something. 


two girls 


Support for 
problem- 
solving 


Everything in there 
(questions, interface 
elements, activities) 
should have a sound 
pedagogical reason 


more coherent, less 
accidental, better 
learning environment 


VE (whole) The equation didn't seem 
necessary. 

FE (min 5-6, 17-18, 20-22) The series of 
buttons allowed them to make judgments 
and adjustments before releasing. 

FE (whole) Because the last question was 
about numerator and denominator, forcing 
them to type in only standard fractions was 
okay. 

VNE (whole) The decimal new diameter 
had no good purpose and wasn’t used. 


Fish 1, 
two boys 



The videotape data were first reviewed by five of the researchers in one group, in 
successive 3-minute segments in order to provide evidence for the design principle being 
effective. During this time, we paused to write down everything we thought relevant after 
each segment and to discuss our observations. Second, detailed notes of students’ work 
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with the EPoW were compiled in order to evaluate the correspondence between assessed 
IBs and evidence of these effects based on student activity (see examples in Table 3.) 
After we watched a student work session, we compared the codes and evidence generated 
independently by five members of the research team. For each DP, the codes were 
shared, compared, and discussed until consensus was reached. Within each video 
analyzed, evidence was provided for almost all DPs. In addition, all four codes (FE, FNE, 
VE, VNE) were evident in the analysis of the video segments. 

We had conversations about the role of student data in our design principle work. Since 
the data was not obtained from studies that were designed to validate the design 
principles, the contribution of the data is to give examples, both pro and con, for specific 
principles. From these, hypotheses can be generated from which future studies can be 
designed. 

6 The Design Principles 

After we generated, refined, and organized the design principles, we came up with a list 
of 23 principles, organized into three categories. Each design principle is listed with 
intended effects, as described in the student data section above. The resulting categorized 
list is located in Appendix B. 

6.1 Three Categories 

During our last session working with the design principles (DPs), we identified three 
categories that the design principles fall under: Motivation, Presentation, and Support for 
Problem-Solving, each of which is described below. During the categorization process, 
we collapsed some of the DPs into others, resulting in a new total of 23 principles. The 
categorized list, along with intended effects, is located in Appendix B. As mentioned 
previously, there were a number of principles that fell under the realm of User Interface 
design, and we did not address those in our list since they are covered extensively in the 
literature. 

Motivation: These design principles promote motivation, including staying on task, 
showing excitement about the process, etc. They include such principles as familiar 
problem context and enabling early reward for students. This category has four design 
principles. 

Presentation: The simplest way to think about these design principles is in terms of 
proofreading for the intended audience. In a sense, this is the counterpart to the “Ease of 
Applet Use” category for everything other than the applet. Some principles that are 
addressed are clarity of the context and the questions and the use of professional 
conventions. Some principles get at applet implementation issues when they’re not about 
the use of the applet, but about the meaning of the things in it. For example, the linked 
representations need to be obvious, or draw attention only to things that support the 
problem solving. The effect we expected was that the understanding of the problem and 
all its facets not be impaired. This category has seven design principles. 
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Support for Problem-Solving: A plurality of the principles falls into this category with its 
12 design principles. All these design principles are intended to facilitate problem 
solving, including things like allowing multiple solution paths, multiple entry points, 
appropriate feedback, and rewarding strategic thought. 

6.2 Interactions between Principles 

These design principles do not stand alone, as each EPoW uses a number of design 
principles. Interactions between the design principles must be considered in the design or 
in the selection of educational software. 

Toward this end, one of our goals is to have a ranking of important design principles. 
While evaluating the design principles that capture the EPoWs, these interactions were 
noted. These are some examples of how some design principles are more important than 
others, though the ordering may not be clear all the time: 

1. User expectations of artifact and interface design are met vs. Attention is drawn to 
the important information. Fish2 was designed so that as each fish is scooped out of 
a pond, a running total is incremented, and a pie chart updated to reflect the ratio of 
males to females that have been selected. As was documented with students using 
Fishl, students expect the updating to occur. However, the fish are constantly 
moving around the pond using animation, and this distracts students from seeing the 
two representations that show the mathematics of what is occurring, which can 
subsequently affect their problem solving ability. 

2. Graphics are great vs. Attention is drawn to the important information. In Hispaniola 
a graphic artist developed pictures and animation that help the student fill cups with 
water, according to the constraints described in the problem. However, students have 
to pay a lot of attention to the moving around of cups to the spigot and furmel, and 
therefore cannot pay a lot of attention to the history list that shows them what they 
have achieved and what they still have to achieve. 

3. History of actions vs. Follow conventions. In Fish2 a history of actions is recorded, 
which helps students see trends in their selections. However, you must remember to 
save in order to obtain a history. The latter goes against what one would expect in the 
interface, given that the history is very helpful in solving the problem. 

6.3 Illustrations 

In addition to the previous videos used for empirical validation, four video tapes of eighth 
grade students working in pairs with a pre-service teacher on the Fish Farm problem 
(Stohl, under review) were analyzed from the perspective of students’ problem solving 
(Stohl & Hollebrands, in progress). The hypothesized design principles (DP) that support 
problem solving and their intended effects (EEs) were used to gauge whether observed 
actions and effects aligned with or contradicted the hypothesized EEs. The designers of 
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the Fish Farm EPoW followed many of the DPs related to problem solving (e.g., allowing 
multiple entry points, supports multiple approaches and solution strategies, uses dynamic 
multiple representations); however, several DPs were not followed (e.g., history of 
actions, programming of applet supports level of accuracy necessary). To illustrate the 
four coding categories (FE, FNE, VE, VNE), consider the followed DP of “uses dynamic 
multiple representations” and the violated DP “history of actions.” 

Multiple Representations. Fish Farm uses multiple representations to provide a visual 
display of male and female fish. As the iconic fish are dragged and dropped in the ponds, 
the ratio count and pie graph displays are dynamically updated. The general IE for this 
DP is to help students: 1) develop representational fluency, 2) facilitate better 
understanding of the problem, and 3) be engaged in mathematical thinking. The ratio 
count and pie graph are intended to display the current status of the part-part and part- 
whole relationship between males and females in the pond that can facilitate a better 
understanding of the problem and engage students in thinking about how to adjust their 
strategy for distributing fish. In addition, it is intended that the ratio counts can alleviate 
having students count fish in the ponds and to promote a transition between reasoning 
part-part and part-whole about the ratios. 

Across these four videos, there were examples where students’ observed actions and 
effects of these actions were aligned (FE) and not aligned (FNE) with the IE for multiple 
representations. One pair of students established the link between the representations and 
the number of fish in a pond early on and subsequently did not have to count the number 
of fish in the pond (FE). With prompting from the pre-service teacher, these same 
students also made connections between the ratio and the pie graph and were able to 
connect the part-part ratio to the idea of a fraction in the pie graph (FE). Another pair of 
students mistakenly reversed the ratio of 1 male to 2 female for Gar’s pond and added 2 
males and 1 female to Gar’s pond (see Figure 3) without apparently using the ratio count 
and/or pie graph to notice that the 3:3 was incorrect (FNE). However, because many 
students initially anticipate a ratio of 1 :2 to result in a pie graph that shows V 2 red and Vi 
yellow, it is possible they used the pie graph accordingly with that expectation and did 
not make the connection between the 3:3 ratio count, the pie graph, and the 1 :2 expected 
ratio displayed for Gar’s pond. It is also appears that they did not notice the difference in 
the pie graphs for Gar’s pond from the before state (figure 3 A) and the after state (figure 
3B). Later in the session, these same students reset the applet to try to find a second 
solution. However, they forget to press the “Run” button and when they placed one male 
and one female fish in Angel’s pond, the representations did not display. The students 
pointed to the pie graph and the ratio count and asked why the displays were not there. 
This provides evidence that the students expected the representations to be displayed and 
were in the habit of attending to these displays (FE). Yet another group seemed not to use 
the representations in their problem solving (FNE). They seemed to focus on the static 
fish in the tank and often counted the swimming fish in the ponds and occasionally 
referred to the updating ratio counts (FE). These students made no reference to the pie 
graph or connections between the fish, ratio and pie graph (FNE). However, these 
students successfully solved the problem and their non-use of the multiple representations 
did not seem to hinder their problem solving (FE). 
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A: All ponds have correct ratio. Gar’s 
pond currently has 1 :2 



B: After placing the reversed ratio of 2:1 
in Gar’s pond, ratio count now 3:3 



Figure 3: Consecutive states of the Fish Farm EPoW. 



History of Actions. Fish Farm did not include a History of Actions feature to keep track 
of students’ correct or incorrect solutions and actions. It was hypothesized that a History 
of Actions would encourage student reflection, strategy tuning, and reduce duplication of 
incorrect solution strategies. For three of the student pairs, the absence of a History of 
Actions seemed to hinder their problem solving (VNE). All three pairs were able to find 
one solution to the problem, but when they were asked to find a different second solution 
they had difficulty remembering their first solution. Recalling the first solution was 
necessary for students to determine if their second solution was different. For example, 
one pair of students quickly found their first solution to the problem by placing 3 male 
and 3 female fish in Angel’s pond, 6 male and 2 female fish in Molly’s pond, and 4 male 
and 8 female fish in Gar’s pond. When challenged to generate a second solution, these 
students made progress but repeated what they had done the first time. It was the pre- 
service teacher, rather than the students, who recalled that their current strategy would 
result in the same solution. However, for another pair of students the absence of a History 
of Actions did not seem to impede their work at all (VE). However, this pair used paper 
to record their first solution and they were able to find a second solution without relying 
on the pre-service teacher to recall what they did the first time. These examples could 
provide evidence to support the argument that memory aids (pre-service teacher, paper 
and pencil. History of Actions) can assist students in solving problems. 
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Appendix A: Highest and Lowest Ranked EPoWs 



Number of Rankings as 


Name (Chronological order) 


Highest Lowest 


Pirates & Diamonds 


111 


Scale n Bowl 


11111 


Llamas 


11 


Llamas 2 


Rock, Paper, Scissors 


mil 


Rock, Paper, Scissors 2 


Earthquake 1 


Earthquake 2 


Earthquake 3 


111 


Earthquake 4 


111 1 


Shoelaces 


11 


Shoelaces 3 1 1 


Search and Rescue (year 1) 


mil 


Elephant 


111 


Hispaniola 


11 


Galactic Exchange 


1111111 


Marathon Graphing 


111 


Scale n Pop 1 1 


Search and Rescue 


111 1 


Graph Zooming 


Fish 


11 


Fish 2 


mil 


Fractris 


nil 


Marabyn 


mil 


Pythagoras’ Mystery Tablet 


1 111 
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Appendix B: The Categorized Design Principles 



Category 


Design Principle 


Intended Effect 


Motivation 


1 . Familiar problem context 


motivation 




2. Use second person voice 


immersive, motivating, creates 
ownership for student 




3. Enable early reward for students (e.g. provide 
easy questions or activities they can do 
successfully) 


get involved in the problem that 
leads toward producing a solution 




4. For videogame-like activities, interactivity, 
high-quality graphics, etc. should match user 
expectations for playability 


get game players to take seriously 
and students continue with the 
problem 



Presentation 1 . Question, cover story and/or introduction 

should be clear, unwordy, unsuperfluous 


students get started quickly 
because they know what to do 


2. Proofread text, labels, etc, with target users 
and age range in mind 


reduce distractions or snag, 
increased focus on learning issues 


3. All other things being equal, use professional 
conventions for content domain 


familiarity, enculturation 


4. Make links between representations obvious 
and ungratuitous 


less division of attention, 
understanding relationships 


5. Use high-quality graphics and other media 
(e.g., still graphics, audio, animation) 


better understanding of the 
problem. 


6. Draw attention only to things that support the 
problem solving 


more on task, more focus on 
inportant issues that will help the 
student to solve the problem 


7. Make everything described in the question 
obvious in the applet; align interactive and 
noninteractive parts 


students oriented more quickly. 
The applet supports student 
solutions to the questions. 



Support for 
problem-solving 


1 . History of actions 


can lead to reflection, strategy 
tuning, and not wasteful 
duplication 




2. Everything in there (questions, interface 
elements, activities) should have a sound 
pedagogical reason 


more coherent, less accidental, 
better learning environment 




3. Allow multiple entry points (e.g., ability, 
experiences, preferences, styles...) 


more students might have many 
ways to get started, get involved 




4. The E-POW supports multiple approaches and students can use different strategies 
multiple solution strategies (e.g., questions and/or to solve the problem - more 
applet) students should be able engage in 

mathematical thinking 




5. Use dynamic multiple representations 
appropriately (linked/notlinked, multiple or 
sources of control) 


develop representational fluency, 
single Facilitate movement toward better 
understanding of the problem. 
More students should be able to 






engage in mathematical thinking. 




6. Give students opportunities to make 
predictions, commit to them, and examine 


students may revise their solution 
strategies. Way to make leamable 




outcomes 


moment 



o 
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7. Thoughtful strategic use of the tool should be 
rewarded more than random use 


less try-and-trash, more thinking 


8. Make a pedagogical decision about whether 
closure is needed. 


sense of accomplishment 


9. Applet should give appropriate status feedback appropriate challenge but doesn’t 

(say the right thing at the right time in the right get too far off track. 

way) 


10. Programming of the applet supports the level less wasteful hairsplitting 
of accuracy necessary for problem solving 


1 1 . Make effort involved in an activity 


more likely to stick with the 


proportional to the importance of what is needed 
to solve a problem (aside from programming for 
accuracy) 


problem. Students attend primarily 
on relevant factors. Less busywork 
in the student's mind. 


12. Technology should add value 


technology is an integral and 
essential element in the problem 
solving process. Students use the 
technology as an essential part of 
their problem solving. 
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